• Can the test of the time transformation be ported over from L3? There are no test updates in this PR, which makes me nervous

Fair point. I added imap_processing/tests/hit/test_hit_l2.py::test_transform_to_10_minute_chunks.

• Can you do an end-to-end test of making an L2 with this new code, making an L3 with that output, and comparing to the existing L3? That will I think be the acid test.

I used Claude (sorry for the verbosity) to create a convincing (at least to the untrained eye, like mine) end-to-end test in this repo.

Here's what you need to run (also found in the repo's README.md):

git clone https://github.com/leowerneck/L2_L3_acid_test.git
cd L2_L3_acid_test
./run_test.sh

If you also see "PASS", that would be reassuring. Let me know if there's anything missing from the test that you would like to see.

I'd feel more confident in something where we set up dependency .json and just ran imap_cli to make the files, so we're doing a full end-to-end on the actual files instead of transforming the L2 and just looking at the Python objects. I can take a stab later this afternoon.

imap-data-access download imap_hit_l2_macropixel-intensity-ed398e36-635a606f_20260819_v001.0003.json
#rename imap_hit_l2_macropixel-intensity-test_20260819_v001.0004.json, change minor version to 4
imap-data-access download imap_hit_l3_macropixel_20260819_v001.0005.cdf # for comparison
imap-data-access download imap_hit_l3_macropixel-6bf00ff0-1e59d123_20260819_v001.0005.json
# rename imap_hit_l3_macropixel-test_20260819_v001.0006.json, change minor version to 6, change minor version of input l2 to 4

imap_cli --instrument hit --data-level l2 --descriptor macropixel-intensity --start-date 20260819 --dependency imap_hit_l2_macropixel-intensity-test_20260819_v001.0004.json
python3 ./imap_l3_data_processor.py --instrument hit --data-level l3 --descriptor macropixel --start-date 20260819 --dependency imap_hit_l3_macropixel-test_20260819_v001.0006.json

cdfcompare -nonumber -noetc imap_hit_l3_macropixel_20260819_v001.0005.cdf imap_hit_l3_macropixel_20260819_v001.0006.cdf

Results are good:

Comparing "imap_hit_l3_macropixel_20260819_v001.0005.cdf" with "imap_hit_l3_macropixel_20260819_v001.0006.cdf"

    DIFFERENT value(s) for gEntry number 1 of attribute "Data_version"
    DIFFERENT number of elements for gEntry number 1 of attribute "File_naming_convention" (40 vs. 45)
    DIFFERENT value(s) for gEntry number 1 of attribute "Generation_date"
    DIFFERENT value(s) for gEntry number 1 of attribute "Logical_file_id"
    DIFFERENT value(s) for gEntry number 1 of attribute "Parents"
    DIFFERENT number of elements for gEntry number 1 of attribute "ground_software_version" (42 vs. 41)
    DIFFERENCES in 15 records for zVariable "epoch" (15 values total)

In these 15 records, the old epoch was 1 microsecond later (greater) than the new epoch. So there's probably some rounding going on involving use of datetimes. I generally prefer to do the math straight in TT2000 for that reason, but I don't think it's important in this case. However it might impact IMAP-Science-Operations-Center/imap_L3_processing/issues/121

The difference in File_naming_convention is apparently because the Great Renaming did not change that attribute on the existing files.

I created this repo which contains a helper to run the test you suggested in an automated way. This was a great exercise to understand how the pipeline runs, so thanks for the example!

Is updating the L2 file required by the Hit integration test (see imap_L3_processing::#161) something I can do, or is something I need to ask someone else to do?

Is updating the L2 file required by the Hit integration test (see imap_L3_processing::#161) something I can do, or is something I need to ask someone else to do?

I think you should be able to do that. If you can chase down the dependency file used for that L2 and just run it again with the new code, that might be most straightforward.

Any thoughts on the microsecond offset? If it's possible to easily squash that it would give me warm fuzzies, but probably not essential.

Any thoughts on the microsecond offset? If it's possible to easily squash that it would give me warm fuzzies, but probably not essential.

The mismatch is explained by how the midpoint is computed. The old L3 code averaged datetime objects, so it rounded each 1-minute epoch to microseconds before averaging. The new L2 code does the math in full-precision TT2000 nanoseconds and only rounds at the very end.

I can change the new L2 code to match the old L3 code, but I just want to double check that the "round-then-average" approach in the old L3 (versus the "average-then-round" in the new L2) is the expected one.